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Abstract: 

A recognition tool according to the invention intelligently recognizes natural input 
before it is passed to a destination or target application. More particularly, the 
recognition tool according to the invention provides better formatting for text 
recognized from natural input, based upon the context in which the text is being 
inserted into a target application. The recognition tool also provides various tools 
for correcting inaccurately recognized text. For example, the recognition tool 
allows a user to select only a part of an inaccurate text, and then identifies 
alternate text candidates based upon the selected portion of the inaccurate text. 
Further, when the user selects text containing multiple words for correction, the 
recognition tool provides cross combinations of alternate text candidates for the 
user's selection. Still further, if the user replaces inaccurate text by submitting a 
new natural input object, the recognition tool ensures that the text recognized 
from the new natural input object is different from the inaccurate text been 
replaced. The recognition tool additionally improves the recognition experience 
after recognized text has been provided to the target application. The recognition 
tool provides the target application with the original natural inpu 
322 

t object for the recognized text, along with the alternate text candidates for that 
original natural input object. Thus, the target application can use the alternate 
text cabinets to correct inaccurately recognized text. Further, a user can inserts 
the original natural input object for recognized text within the target application. 
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(54) Natural input recognition tool 

(57) A recognition tool according to the Invention in- 
telligently recognizes natural input before it is passed to 
a destination or target application . More particularly, the 
recognition tool according to the invention provides bet- 
ter formatting for text recognized from natural input, 
based upon the context In which the text Is being Insert- 
ed into a target application. The recognition tool also 
provides various tools for correcting Inaccurately recog- 
nized text. For example, the recognition tool allows a 
user to select only a part of an Inaccurate text, and then 
Identifies alternate text candidates based upon the se- 
lected portion of the inaccurate text. Further, when the 
user selects text containing multiple words for con-ec- 
tion, the recognition tool provides cross combinations of 



alternate text candidates for the user's selection. Stiii 
further, ifthe user replaces inaccurate text by submitting 
a new natural Input object, the recognition tool ensures 
that the text recognized from the new natural input ob- 
ject is different from the Inaccurate text been replaced. 
The recognition tool additionally Improves the recogni- 
tion experience after recognized text has been provided 
to the target application. The recognition tool provides 
the target application with the original natural input ob- 
ject for the recognized text, along with the alternate text 
candidates for that original natural input object. Thus, 
the target application can use the alternate text cabinets 
to correct inaccurately recognized text. Further, a user 
can inserts the original natural input object for recog- 
nized text within the target application. 
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Description 

Field of the Invention 

5 [0001] The present Invention relates to a recognition tool for Improving a user's experience In recognizing text from 
natural input, such as speech or handwriting Input. More particularly, the present invention relates to a recognition tool 
that Improves the user's recognition experience both before and after text recognized from a natural Input object Is 
inserted into a target application. 

10 Background of the Invention 

[0002] As computers become more and more Integrated into our modem society, there Is a growing need for tools 
that will allow a user to efficiently enter data Into a computer. Some of the most desired Input tools are those that allow 
a user to enter data using natural input techniques rather than typing on a physical or software QWERTY Iceyboard. 

IS These natural input techniques, which include handwriting and speech, offer users the promise of a familiar and con- 
venient method of creating data that requires no special skills. To this end, great strides have been made in developing 
applications that recognize text from such natural input. For example, some voice recognition applications will accu- 
rately transcribe text from a user's speech. Similarly, some handwriting recognition applications will recognize text from 
both block print and cursive handwriting. 

20 [0003] One important advantage of these natural input techniques is that they potentially offer users a much higher 
throughput than entering data through a keyboard. The typical user can speak considerably faster than he or she can 
type. Thus, a user can quickly Input text by speaking that would otherwise take much longer to Input by typing on a 
l<eyboard. Likewise, if a user has not memorized the position of keys on a QWERTY keyboard, that user may be able 
to handwrite words with a stylus and a digitizer faster than he or she can type those words on a keyboard. 

25 [0004] While natural Input offers the potential for high throughput in theory, in actuality the inherent inaccuracies of 
natural input can significantly reduce that potential. For example, the voice recognition application may be unable to 
distinguish the spoken word "they're" from "there." A handwriting recognition application may similarly be unable to 
distinguish the handwritten word "clog" from the handwritten word " dog." Cuaently, natural input recognition engines 
provide recognition candidates based only upon the natural input object to be recognized. The destination or intended 

30 use of the recognized text has no influence over how a recognition engine selects a recognition candidate to recognize 
a natural input object. 

[0005] In addition to the difficulties in accurately recognizing text from a natural input object, there are additional 
difficulties in con-ectly formatting the recognized text. For example, it is difficult for a natural input recognition engine 
to detenmine if recognized words should have spacing between them, and, if so how many spaces. Other than pauses, 

35 speech input provides no information as to spacing. While handwriting has some mechanisms for indicating spacing, 
such as proofing marks and leaving space between handwritten words, these cues are cumbersome for the user to 
employ or difficult for a recognition engine to accurately interpret. Stmilariy, it may be difficult for a natural input recog- 
nition engine to detennine whteh characters in recognized text should be capitalized, if any. Still further, it may be 
difficult for a natural input recognition engine to properly recognize punctuation from a natural input object. 

40 [0006] In an attempt to address this problem, some recognition engines include an automatic spacing feature that 
automatically provides spacing for recognized text. The solution has not wori<ed very well, however. For example, if 
an automatic spacing process always inserts a space after recognized text, then when the user inserts a connection 
into an existing word, the automatic spacing process will create and unwanted space In the middle of the resulting 
word. On the other hand, when the user adds a word at a location where a space already exists, the automatic spacing 

45 process will create an extra, unwanted space. 

Summary of the Invention 

[0007] To address these problems, the recognition tool according to the invention intelligently recognizes natural 
50 Input before it is passed to a destination or target applrcation. More partlculariy, the recognition tool according to the 
Invention provides better formatting for text recognized from natural input, based upon the context in which the text Is 
being Inserted into a target application. In addition, the recognition tool creates a model object that includes the altemate 
text candidates identified for each natural input object in a document. When a natural input recognition engine is indif- 
ferent to selecting between multiple alternate text candidates to recognize a natural input object, the recognition engine 
55 can create a lattice of the different text candidates according to, for example, a natural language model. If a user then 
definitively selects a text candidate corresponding to one natural Input object, the lattice can be used to con^ectly select 
text candidates for other natural input objects. 

[0008] The recognition tool according to the invention also provides various tools for correcting inaccurately recog- 
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nlzed text. For example, the recognition tool allows a user to select only a part of an Inaccurate text, and then Identifies 
alternate text candidates based upon the selected portion of the inaccurate text. Further, when the user selects text 
containing multiple words for correction, the recognition tool provides cross combinations of alternate text candidates 
for the user's selection. Still further, if the user replaces inaccurate text by submitting a new natural input object, the 
5 recognition tool ensures that the text recognized from the new natural input object is different from the inaccurate text 
been replaced. 

[0009] In addition to improving the user's recognition experience before text is sent to the destination or target soft- 
ware application, the recognition tool of the invention also improves the user's recognition experience after recognized 
text has been sent to the target application. The recognition tool provides the target application with the original natural 
10 input object for the recognized text, along with the alternate text candidates Identified for that original natural input 
object. Thus, the target application can use the alternate text candidates to correct inaccurately recognized text. Further, 
a user can insert the original natural input object from which text was recognized into the target application. 

Brief Description of the Drawings 

15 

[0010] The foregoing summary of the invention, as well as the following detailed description of prefen-ed embodi- 
ments, will be better understood when read in conjunction with the accompanying drawings. 
[0011] Figure 1 shows a schematic diagram of a general-purpose digital computing environment that can be used 
to implement various aspects of the invention. 
20 [0012] Figure 2 illustrates components of a recognition tool according to an embodiment of the invention. 

[0013] Figures 3A and SB illustrate flowcharts showing a method of operation for the recognitiori tool according to 
an embodiment of the invention. 

[001 4J Figure 4 illustrates one embodiment of a correction interface that the recognition tool of the invention may 
provide for correcting inaccurately recognized text. 
25 [001 5] Figure 5 illustrates another embodiment of a correction interface that the recognition tool of the invention may 
provide for correcting inaccurately recognized text. 

[0016] Figure 6 illustrates components of a recognition tool according to an embodiment of the invention. 
Detailed Description of Preferred Embodiments 

30 

Overview 

[0017] The present Invention relates to a recognition tool for more accurately recognizing and formatting text from 
natural input. A recognition tool according to the Invention may be implemented with instructions that can be executed 
35 on a computing device. Accordingly, an exemplary computing environment for executing such instructions will be de- 
scribed, The components and functions of a recognition tool according to the invention will be described in detail as 
well, along with a correction interface that maybe employed by the recognition tool according to the Invention. 

Exemplary Operating Erwiror^ment 

40 

[0018] As previously noted, the text recognition tool of the invention may be implemented using software. That is, 
the recognition tool may be described in the general context of computer-executable instructions, such as program 
modules, executed by one or more computing devices. Generally, program modules Include routines, programs, ob- 
jects, components, data structures, etc. that perfomi particular tasks or implement particular abstract data types. Typ- 

45 icaily the functionality of the program modules may be combined or distributed as desired in various embodiments. 
[0019] Because the invention may be implemented using software, it may be helpful for a better understanding of 
the invention to briefly discuss the components and operation of a typical programmable computer on which various 
embodiments of the invention may be employed. Such an exemplary computer system Is Illustrated In Figure 1 . The 
system includes a general-purpose computer 100. This computer 100 may take the forni of a conventional personal 

50 digital assistant, a tablet, desktop or laptop personal computer, network server or the like. 

[0020] Computer 1 00 typically includes at least some form of computer readable media. Computer readable media 
can be any available media that can be accessed by the computer 1 00. By way of example, and not limitation, computer 
readable media may comprise computer storage media and communication media. Computer storage media includes 
volatile and nonvolatile, removable and non-removable media implemented In any method or technology for storage 

55 of infomiation such as computer readable Instructions, data structures, program modules or other data. Computer 
storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, 
CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk 
storage or other magnetic storage devices, or any other medium whteh can be used to store the desired infomiation 



3 



EP 1 365 349 A2 



and which can be accessed by the computer 1 00. 

[0021] Communication media typically embodies computer readable Instructions, data structures, program modules 
or other data In a modulated data signal such as a carrier wave or other transport mechanism and Includes any Infor- 
mation delivery media. The term "modulated data signal" means a signal that has one or more of its characteristics 
5 set or changed In such a manner as to encode Information In the signal. By way of example, and not limitation, com- 
munication media includes wired media such as a wired network or direct-wired connection, and wireless media such 
as acoustic, RF, infrared and other wireless media. Combinations of any of the above should also be included within 
the scope of computer readable media. 

[0022] The computer 1 00 typically Includes a processing unit 1 1 0, a system memory 1 20, and a system bus 1 30 that 
10 couples various system components including the system memory to the processing unit 110. The system bus 130 
may be any of several types of bus structures Including a memory bus or memory controller, a peripheral bus, and a 
local bus using any of a variety of bus architectures. The system memory 120 includes read only memory (ROM) 140 
and random access memory (RAM) 150. A basic input/output system 160 (BIOS), containing the basic routines that 
help to transfer infomnation between elements within the computer 100, such as during start-up, is stored in the ROM 
15 140. 

[0023] The computer 1 00 may further include additional computer storage media devices, such as a hard disk drive 
170 for reading from and writing to a hard disk, a magnetic disk drive 180 for reading from or writing to a removable 
magnetic disk 190, and an optical disk drive 191 for reading from or writing to a removable optical disk 192, such as 
a CD ROM or other optical media. The hard disk drive 170, magnetic disk drive 180, and optical disk drive 191 are 
20 connected to the system bus 1 30 by a hard disk drive interface 1 92, a magnetic disk drive interface 1 93, and an optical 
disk drive interface 194, respectively. The drives and their associated computer-readable media provide nonvolatile 
storage of computer readable instructions, data structures, program modules, and other data for the personal computer 
100. 

[0024] Although the exemplary environment described herein employs a hard disk drive 1 70, a removable magnetic 

25 disk drive 180 and a removable optical disk drive 191 , it should be appreciated by those skilled In the art that other 
types of computer readable media which can store data that is accessible by a computer, such as magnetic cassettes, 
flash memory cards, digital video disks, Bernoulli cartridges, random access memories (RAMs), read-only memories 
(ROMs) and the like may also be used in the exemplary operating environment. Also, It should be appreciated that 
more portable embodiments of the computer 100, such as a tablet personal computer or personal digital assistant, 

30 may omit one or more of the computer storage media devices discussed above. 

[0025] A number of program modules may be stored on the hard disk drive 1 70, magnetic disk 1 90, optical disk 1 92, 
ROM 140, or RAM 150, including an operating system 195. one or more application programs 196, other program 
modules 197, and program data 198. A user may enter commands and Infomiatlon into the computer 100 through 
various input devices, such as a keyboard 101 and a pointing device 102 (for example, a mouse, touchpad, or pointing 

55 stick). As previously noted, the invention is directed to a correction tool that Is very useful forcorrecting text inaccurately 
recognized from a natural input technique. Accordingly, the computing device 120 will typically also include input de- 
vices for accepting natural input. For example, the computer 100 may include a digitizer 165 and a stylus 166, with a 
user may employ to create handwriting data. The computer 100 may also include a microphone 167, through which a 
user can input speech infomiation for recognition by a voice recognition process. The computer 100 may also have 

40 additional input devices, such as a joystick, game pad, satellite dish, scanner, touch screen, or the like. 

[0026] These and other input devices often are connected to the processing unit 1 1 0 through a serial port interface 
106 that is coupled to the system bus 130. but may be connected by other interfaces, such as a parallel port, game 
port, or a universal serial bus (USB). Further still, these devices may be coupled directly to the system bus 1 30 via an 
appropriate interface (not shown). A monitor 107 or other type of display device is also connected to the system bus 

45 1 30 via an interface, such as a video adapter 1 08. In addition to the monitor 1 07, personal computers typically include 
other peripheral output devices (not shown), such as speakers and printers. As will be appreciated by those of ordinary 
skill in the art, the monitor 1 07 may incorporate the digitizer 1 65. This arrangement conveniently allows a user to employ 
the stylus 1 66 to point directly to objects displayed on the monitor 1 07. 

[0027] The computer 1 00 may operate In a networked environment using logical connections to one or more remote 
50 computers, such as a remote computer 109. The remote computer 109 may be a server, a router, a network PC, a 
peer device, or other common networi< node, and typically includes many or all of the elements described above relative 
to the computer 100, although only a memory storage device 111 with related applications programs 196 have been 
illustrated in Figure 1 . The logical connections depicted In Figure 1 include a local area network (LAN) 112 and a wide 
area network (WAN) 113. Such networking environments are commonplace in offices, enterprise-wide computer net- 
55 wori(s, intranets, and the Internet, and thus will not be explained In detail here. 

[0028] When used In a LAN networking environment, the computer 1 00 is connected to the local networi< 1 1 2 through 
a network interiace or adapter 114. When used in a WAN networicing environment, the personal computer 1 00 typically 
includes a modem 1 1 5 or other means for establishing a communications link over the wide area networi< 1 1 3, e.g. , to 
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the Internet. The modem 1 1 5, which may be Internal or external, may be connected to the system bus 1 30 via the serial 
port Interface 106. In a networked environment, program modules depicted relative to the personal computer 100, or 
portions thereof, may be stored in a remote memory storage device. Of course, it wilt be appreciated that the network 
connections shown are exemplary and other techniques for establishing a communications link between the computers 
s may be used. The existence of any of various well-known protocols such as TCP/I P, Ethernet, FTP, HTTP and the like 
IS presumed, and the system may be operated In a client-server configuration to permit a user to retrieve web pages 
from a web-based server. Any of various conventional web browsers may be used to display and manipulate data on 
web pages. 

10 The Recognition Tool 

[0029] Turning now to Figure 2, this figure illustrates the components of a recognition tool according to one embod- 
iment of the invention, together with the components of the environment in which it operates. More particularly, this 
fligure Illustrates a tablet input panel 201, the natural Input recognition module 203 according to the invention, a text 

15 services framework module 205 and a target application 207. In order to provide a better understanding of the Invention , 
the environment in which the recognition tool of the Invention (that is, the natural input recognition module 203) operates 
will first be described. The components and operation of the natural Input recognition module 203 will then be discussed. 
[0030] As will be appreciated by those of ordinary skill In the art, the tablet Input panel 201 is a tool that allows a 
user to submit one or more types of natural input for recognition and delivery to the target application 207. For example, 

20 the tablet Input panel 201 may provide a user interface for receiving handwriting input, both for recognizing script and 
for recognizing individual characters. Alternately, or additionally, the tablet input panel 201 may provide a user Interface 
for initiating and stopping a voice recognition process. It may also provide still other techniques for submitting Input 
data, such as displaying a soft keyboard. Further, the tablet Input panel 201 may provide a display, sometimes refen-ed 
to as a "stage," for previewing text that has been recognized from natural Input before the text Is Inserted into the target 

25 application 207. in the illustrated embodiment, the tablet input panel 201 provides user interfaces for receiving both 
speech and handwriting input. 

[0031] Turning now to the text services framework module 205, this module may be implemented using the text 
services framework tool currently provided by Microsoft Corporation In the Microsoft Windows® XP operating system. 
As known to those of ordinary skill in the art, this text services framework tool serves as a generic input device for 
30 inputting text into a compatible target application from a variety of text sources, Including the tablet input panel 201 . 
In addition, when the target application supports all of the functions offered by the text services framework tool, the 
text services framework tool can obtain context Infonnation from the target application. More particularly, the text serv- 
ices framework tool can ascertain a variety of types of context infonnation regarding the location In the target application 
207 at which It will Insert text. 

35 [0032] This context Infomnation may Include, for example, the text, If any, that already exists on either side of the 
insertion location. Further, it may include the length of the insertion location (the insertion location may be a single 
insertion point, for example, or a selection of existing text). It may also include any existing text that will be replaced 
by the insertion of new text at the insertion location. One partbularly useful type of context information that may be 
obtained by the text services framework tool Is the type of field in which the insertion location is positioned. More 

40 particularly, the field type for a field, conventionally refered to as a "factoid," describes the type of data that should be 
Inserted into that field. For example, the address field for a browser application will typically have the field type or 
factoid "WEB," because the only data that should be inserted into the address field will take the fomn of "http://www. 
microsoft.com," 

[0033] Similarly, a field In a web site prompting a user for his or her e-mail address will have the field type or factoid 
45 "EMAIL,** because the only data that should be inserted into this field will take the fomn of "someone® example.com." 
It should be noted that, in both of these examples, the field type inherently indicates that no spaces should occur in 
the text inserted into these fields. A complete listing of field types and combinations of field types currently supported 
by the text services framework tool in the Mtorosoft Windows® XP operating system are listed below in Table 1 and 
Table 2, respectively. 

50 



Table 1 







Factoids 


Definition/Examples 


NONE 


Disables all other factoids 


SYSDICT 


Enables the system dictionary only 



5 



10 



15 



20 



25 



30 



35 



40 



45 



50 
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Table 1 (continued) 



Factoids 


Definition/Examples 


WORDLIST 


Enables the word list only 


EMAIL 


someone® exanriple.coni 


WEB 


http://www.microsoft.com 


ONECHAR 


All isolated ANSI char 




supported 


NUMBER 


All num classes 


DIGIT 


0-9 


CURRENCY 


$45 60 £50 ¥30 33 


POSTAL CODE 


Postal codes 981 1P 


PERCENT 


87% 


DATE 


10/30/2001 '01 31/12 12/9Q 1999-2000 


TIME 


12'23'00 PM 12*30 24*30 12*2*^01 1*12 AN/I 


TELEPHONE 


123 555 0190 0-123-206 555 0190 ^206^555-0190 

1 www V 1 </ W f V 1 ^\J\J www \J 1 WV| t^WU/WWW W 1 WW 


FILENAME 


mvdocunient doc O'XmvfolddAfllA r 

1 1 ly UwwUI 1 10 1 l&iUWw| Wi V 1 ly 1 wivlwl U IIO«w 


UPPERCHAR 




JPN COMMON 


Commonlv ii^iprl Kflnii Katakflnfl And Himnpna phflmrtArQ 

w 1 1 II J IVy 1 1 1 y ^JO^vJ I\CIMJI| 1 \Cll.Cir\m iClf OIIVJ l IllCl^miCI wllulOwlwIw 


CHS COMMON 


Commonh/ used Kanii Katakann and Hiranana nharnotAre 

^'wi 1 II I iwi iiy Mwcw r\ciiiji| rvoiidivcii ici| miu i iiiauciiio wiiciiawlwio 


CHT COMMON 


r^nmmon Kanli Katakflnn and Miranano nhnrartorc 
wv^i 1 II 1 iui 1 r\ciiiji| rvdioivcii Id) aiiu riiici^ciiici v^iiciictwicio 


KOR COMMON 


Commonlv used Kanii Katakana and Hiranana nharantprc: 
wwi 1 II 1 lui iiy uocij r\ciiiji| r\aicii\ciiici| ciiiu riiici^ciiici wllciiciwlCrlo 


HIRAGANA 


Hlragana characters only 


KATAKANA 


Katakana characters only 


KANJLCOMMON 


Commonly used Kanji characters 


BOPOMOFO 


Bopomof 0 characters 


JAMO 


Hangul compatibility Jamo characters based 


HANGUL_COMMON 


Commonly used Hangul characters 


Table 2 



Factoid Combination 


Definition 


WEB i WORDLiST 


Web factoid or the word list 


EMAIL 1 WORDLIST 


E-mail factoid or the word list 


FILENAME i WEB 1 WORDLIST 


File name factoid or Web factoid and the word list 



[0034] Of course, those of ordinary skill In the art will appreciate that the each field type definition listed above may 
be expanded or narrowed, as desired. It should also be appreciated that various embodiments of the invention may 
employ additional or alternate field types than those listed above, and that the number of field types that can be Identified 
by the text services framework module 205 can vary from the list above. Further, any other suitable tool than the text 
services framework tool provided in the Microsoft Windows® XP operating system can be employed as the text services 
framework module 205. For example, a tool could be employed that only obtains context infomr^ation from the target 
application 207, and does not provide a variety of features for inserting text from multiple sources in to the target 
application 207. 

[0035] With regard to the target application 207, It may be any application that supports the text services frameworic 
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module 205 by providing context information to the text services framework module 205. For example, the target ap- 
plication 207 may be the Microsoft Word XP word-processing software product, in this example, the text recognized 
from natural input objects submitted to the tablet input panel 201 is Inserted through the text services framework module 
205 Into a document being edited by the Microsoft Word XP word processing software product. 

s [0036] To support the text services framework module 205, the target application 207 will include a text services 
framework interface 209. This interface 209 collects and provides the context information to the text services framework 
module 205. As previously noted, for example, the text services framework Interface 209 can identify the text adjacent 
to an insertion location, it can also identify the length of the insertion location, the text, if any that will be replaced by 
text inserted at the Insertion location, and the type of field in which the insertion location is positioned. Also, as will be 

10 discussed In detail below, the text sen/ices framework Interface 209 can store a variety of text infomnation provided by 
the text sen^lces framework module 205. 

[0037] Turning now to the natural input recognition module 203, the natural Input recognition module includes a 
recognition context module 211 , a recognition module 213, an object model module 21 5, and a correction module 21 7. 
Each of these modules 211-217 wilt now be discussed in turn. 

IS [0038] The recognition context module 211 provides context information to the recognition module 213 regarding 
the insertion of text recognized by the recognition module 213 from natural input. This context information allows the 
recognition module 213 to intelligently recognize text from natural input based upon the destination of the recognized 
text. For example, the recognition context module 211 receives the context infonnation identified through the text serv- 
ices framework module 205. From this context infomriation, the recognition context module 211 categorizes the field 

20 in which the Insertion location Is positioned, in the illustrated example, the recognition context module 211 will categorize 
a field In the target application 207 into one of three different categories. 

[0039] The first category of fields are "controlled" fields that fully support the text services framewort< module 205. 
Thus, afield in this category will provide the text services framework module 205 with all applicable context infonnation. 
Further, the text services framework module 205 can control the content of the field, so that the text services framewori< 
25 module 205. can intelligently Insert text into the field. As will be discussed below, by controlling the content of this 
category of field, the text services framework module 205 can be used to implement automatic spacing heuristics that 
delete existing spaces from a controlled field, in addition to adding space. 

[0040] The next category of field is the "smart" field. This field provides context infonnation to the recognition context 
module 21 1 , but does not offer control to the text services framework module 205. Instead, the text services framewori< 

30 module 205 inserts text to this type of field simply by sending key data. A smart field may provide context information 
to the recognition context module 21 1 through the text services framework module 205, as described above. Alternately, 
the target application 207 may Implement a smart field may using an edit control, rich edit control, or other data structure 
that will allow the recognition context module 211 to directly obtain the field's context infonnation from the target ap- 
plication. The third category of field Is the "dumb" field, which provides no context information to the recognition context 

35 module 21 1 . An example if this category of field is a custom drawn, non -windowed control, such as an edit box drawn 
in a Web page. 

[0041] Using these field categories, the recognition context module 211 may generate fomriatting heuristics. For 
example, the recognition context module 211 may create spacing heuristics based upon the text sun^ounding the in- 
sertion location. For example, with a "controlled" field, the recognition context module 211 may implement the following 
40 automatic spacing rules for text being sent to the target application 207: 

A. Insert a space before a send that starts with a word or a punctuation $, #, (, [, {, but only if: 

There's not already a non-printing character before the insertion location AND 

45 

This text being sent is NOT one of the following, which should concatenate to what precedes them: 

1 . The text is being sent to an insertion location that Is a selection of existing text, 
OR 

so 2. Text being sent to an Insertion location positioned after a designated as one to 'never insert space after': 

®.$. #, (, [. {. --.../Aand -; 
OR 

3. The text is being sent to an insertion location that is within a complete or partial web or file or email 
address (such as when two or more existing contiguous printing characters in the field start with http, ftp: 

55 //, //, \\, www, or c: (or any single letter followed by a colon) or including the character @) 

OR 

4. The text is being sent to an insertion location positioned after a character that is a number and the first 
character in the text being sent is a number 
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OR 

5. The first word in the text being sent has been designated to "remove space before* (such as domain 

names preceded by a period (.com) and Jl 

OR 

5 6. The first word in the text being sent, when combined with a word positioned before the Insertion location, 

creates a word other than a "snicker word," such as "penis," except when the first word in the text being 
sent is 'a' so as not to concatenate the 'a' and word positioned before the insertion location Into a compound 
word. 

7. No space is inserted between a period and a iowercase character. 

10 

B. Remove existing single-line space characters (such as 'hard-tab/ 'space,' and 'non-breaking space') positioned 
before the insertion location for sent text having a first word that is in the iist in A5 above, or a first character that 
is!,?, ©,).],}, .. or -. 

C. insert a space after sent text that ends with a word or a phrase-ending punctuation, such as ),],}, :, but oniy if: 

IS 

There's not already a space after the insertion location 
AND 

The text being sent Is NOT one of the foiiowing, which should concatenate to what follows them: 

20 1 . The text is being sent to an insertion location that is a selection of existing text, 

OR 

2. The ending word In the sent text ends with a character designated as 'never insert space after' (which 

may be the same list as in A2 above) 
OR 

2s 3. The ending word in the sent text is a web or file or email address (which may be identified using the 

same iist as in A3 above) 
OR 

4. The last word being sent when combined with the next word creates a word other than a "snicker 
word," such as "penis". 

so 5. No space is Inserted between a sentence ending punctuation and a lower case character. 

D. Insert one space after a multi-character insertion whose last character is . or I or ? (for example, dont add any 
spaces after a lone period). 

E. Include spaces between words in the sent text, as indicated by an English default language model, but NOT 
35 after a word that includes @ or begins with http://, https://, ftp://, //, W, www, or c: (or any other letter followed by a 

colon). 

F. include the foiiowing prefixes In the natural input text recognition drctionary (that is, the dictionary employed to 
recognize text from natural input objects): //, \\, and ://. 

G. Include the foiiowing suffixes in the natural Inputtext recognition dictionary: ing, 's, er, nl, and typically registered 
40 file extensions both alone and preceded by a period (for example, both exe and .exe), and Internet domains both 

alone and preceded by a period (for example, com and .com). 

H. Include common abbreviations in the natural input text recognition dictionary, such as e.g., i.e., a.k.a., and etc. 

[0042] For smart fields, a similar set of spacing heuristics may be employed. Because these fields do not provide 
45 control to the text services framewori< module 205, however, these heuristics will typically not delete existing spaces, 
to avoid an Inadvertent deletion of space from the wrong location (set forth in step B, above). Thus, the recognition 
context module 211 can Implement the following automatic spacing characteristics for smart fields: 

A. Insert a space before a send that starts with a word or a punctuation $, #, (, [, {, but only if: 

so 

There's not already a non-printing character before the insertion location 
AND 

This text being sent is NOT one of the following, which should concatenate to what precedes them: 

55 

1 . The text is being sent to an Insertion location that is a selection of existing text, 
OR 

2. Text being sent to an insertion location positioned after a character designated as one to 'never insert 
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space after': ©,$,#, (, [, {, /, \ and -; 
OR 

3. The text is being sent to an insertion location that is within a complete or partial web or file or email 
address (such as when two or more existing contiguous printing characters In the field start with http, ftp: 

5 II, II, \\, WWW, or c: (or any single letter followed by a colon) or Including the character @) 

OR 

4. The text is being sent to an insertion location positioned after a character that is a number and the first 
character in the text being sent is a number 

OR 

'0 5. The first word in the text being sent has been designated to 'remove space before' (such as domain 

names preceded by a period (.com) and Jl 
OR 

6, The first word in the text being sent, when combined with a word positioned before the insertion location, 
creates a word other than a "snicker word," such as "penis," except when the first word in the text being 

'5 sent is 'a' so as not to concatenate the 'a' and word positioned before the Insertion location Into a compound 

word. 

7. No space is inserted between a period and a lower case character. 

B. Insert a space after sent text that ends with a word or a phrase-ending punctuation, such as ),],},:.;,", but only if: 

20 

There's not already a space after the insertion location 
AND 

The text being sent is NOT one of the following, which should concatenate to what follows them: 

1 . The text is being sent to an insertion location that is a selection of existing text, 
OR 

2. The ending word in the sent text ends with a character designated as 'never insert space after' (which 

may be the same list as in A2 above) 
OR 

^0 3. The ending word in the sent text is a web or file or email address (which may be identified using the 

same list as In A3 above) 
OR 

4. The last word being sent when combined with the next word creates a word other than a "snicker 

word," such as "penis". 

^5 5. No space is inserted between a sentence ending punctuation and a lower case character. 

0. Insert one space after a multi-character insertion whose last character is . or ! or ? (for example, don't add any 
spaces after a ione period). 

^0 D. Include spaces between words in the sent text, as indicated by an English default language model, but NOT 

after a word that includes @ or begins with http://, https://, ftp://, //, W, www, or c: (or any other letter followed by a 
colon). 

E. Include the following prefixes In the natural Input text recognition dictionary: //, W, and ://. 

45 

F. Include the following suffixes In the natural input text recognition dictionary: ing, 's, er, n't, and typically registered 
file extensions both alone and preceded by a period (for example, both exe and .exe), and internet domains both 
alone and preceded by a period (for example, com and .com). 

50 Q. Include common abbreviations In the natural input text recognition dictionary, such as e.g., i.e., a.k.a., and etc. 

[0043] Simpler automatic spacing heuristics can be employed for "dumb" fields. Because these fields do not provide 
context infomiatlon identifying the text (if any) around the insertion location, and each erroneously created space 
presents a burden for the user to correct, space typically Is only Inserted after sent text. Thus, the context recognition 
55 module 211 can implement the following automata spacing heuristics for such "dumb" fields: 

A. Insert a space aftersenttext that ends with a word or a phrasoTending punctuation, such as ), ], }, :, ;,". but only if: 
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There's not already a space after the insertion location 
AND 

The text being sent is NOT one of the following, which should concatenate to what follows them: 

5 

1 . The text is being sent to an insertion location that is a selection of existing text, 
OR 

2. The ending word in the sent text ends with a character designated as 'never Insert space after*: @, $, 
#. (. [, {.-.-./.V and-. 

10 OR 

3. The ending word in the sent text is a web or file or email address (such as when two or more existing 
contiguous printing characters In the field start with http, ftp://, //, W, www, ore: (or any single letter followed 
by a colon) or including the character @). 

IS B. Insert one space after a multi-character insertion whose last character is . or ! or ? (for example, don't add any 
spaces after a lone period). 

0. Include spaces between words in the sent text, as Indicated by an English default language model, but NOT 
after a word that includes @ or begins with http://, https://, ftp://, //, W, www, or c: (or any other letter followed by a 
colon). 

20 D. Include the following prefixes in the natural input text recognition dictionary: //, W, and J/ , 

E. Include the following suffixes In the natural Input text recognition dictionary: Ing, 's, er, n't, and typically registered 
file extensions both alone and preceded by a period (for example, both exe and .exe), and Intemet domains both 
alone and preceded by a period (for example, com and .com). 

F. include common abbreviations in the natural input text recognition dictionary, such as e.g.. I.e., a.k.a., and etc. 

25 

[0044] Of course, the context Infomriatlon can be used to override or modify the automatic spacing heuristics desig- 
nated for a "controlled" field or "smart" field. For example, if the context recognition module 21 1 detennlnes that a field 
Is for a password. It can prevent the automatic spacing heuristics from being applied to that field. 
[0045] Also, while the above-described automatic spacing rules may be implemented for English-language based 
30 natural input recognition, other rules may be applied for different languages as appropriate. For example, with French- 
language based natural input recognition, the recognition context module 211 may implement the following automatic 
spacing rules for text being sent to a controlled field in the target application 207: 

A. Insert a space before a send that starts with a word or a punctuation I, ?, :, ;, $, #, {, [, {, but only if: 

35 

There's not already a non-printing character before the insertion location 
AND 

This text being sent is NOT one of the following, which should concatenate to what precedes them: 

^0 1 . The text is being sent to an insertion location that is a selection of existing text, 

OR 

2. Text being sent to an insertion location positioned after a character designated as one to 'never insert 
space after*: ©,$,#, (. [, {, _, /, \, and -; 

OR 

3. The text Is being sent to an Insertion location that Is within a complete or partial web or file or email 
address (such as when two or more existing contiguous printing characters in the field start with http, ftp: 
//, //, W, www, or c: (or any single letter followed by a colon) or including the character @) 

OR 

4. The text is being sent to an insertion location positioned after a character that is a number and the first 
so character in the text being sent is a number 

OR 

5. The first word in the text being sent has been designated to "remove space before' (such as domain 
names preceded by a period (.com) and J/ 

OR 

ss 6. The first word in the text being sent, when combined with a word positioned before the Insertion location, 

creates a word other than a "snicker word," such as "penis," except when the first word In the text being 
sent is 'a' so as not to concatenate the 'a' and word positioned before the Insertion location Into acompound 
word. 
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7. No space Is inserted between a period and a lowercase character. 

B. Remove existing single-iine space characters (such as 'hard-tab,' 'space/ and 'non-breaking space') positioned 
before the Insertion location for sent text having a first word that Is In the list In A5 above, or a first character that 
5 Is®. ).],}...:.;. ".or-. 

[0046] In addition to spacing fomiat, the context infomriation also can be used to deternnine capitalization of characters 
in recognized text. For example, if the recognition context module 211 detemriines that the insertion location Is within 
a field of the type "E-MAIL," the recognition context module 21 1 may generate heuristics allowing the recognition module 

10 213 to allow the capitalization of any character In the recognized text. Ordinartty, the recognition module might allow 
only the first letter In a word to be capitalized. If the recognition context module 211 employs the context infonnatlon 
to realize that the recognized text is an e-mail address, however, it can then allow the recognition module 213 to 
capitalize any character in the recognized text. In other situations, the recognition context module 21 1 may instruct the 
recognition module 21 3 not to capitalize any character in a recognized word other than the first character Alternately, 

15 the recognition context module 21 1 may determine that the insertion location Is within a field of the type "UPPERCHAR," 
and thus should only receive data in capitalized characters. The recognition context module 21 1 would then generate 
heuristics requiring the recognition module 213 to capitalize recognized text being inserted into this field. 
[0047] IVIoreover, the recognition context module 211 can generate heuristics instructing the recognition module 21 3 
to provide punctuation for recognized text. For example, if the recognition context module 211 determines that the 

20 insertion location is within a field of the type 'TELEPi^ONE" and thus should receive only data corresponding to tele- 
phone numbers, then the recognition context module 211 can generate heuristics Instructing the recognition module 
213 to provide dashes between groups of recognized text. For example, the recognition context module 211 could 
generate heuristics instructing the recognition module 213 to insert dashes after the third and sixth numbers In recog- 
nized text for this field. Thus, if the user handwrote 1 234567890, the recognition module would insert dashes to generate 

25 the text 123-456-7890. 

[004B] Additionally, the recognition context module 211 may prevent the recognition module 213 from replacing ex- 
isting text with Identical recognized text. For example, a user may be correcting text that already exists in the target 
application 207. In this situation, the user will not want the recognition module 21 3 to generate recognized text identical 
to the existing text that the user intends to correct. By identifying the existing text through the text services framework 
30 module 205, the recognition context module 21 1 can provide instructions to the recognition module 213 that will prevent 
the recognition module 213 from replacing existing text with identical recognized text. 

[0049] Referring now to the recognition module 213, the recognition module 213 generates text from natural input 
objects. More partteutarly, the recognition module 21 3 Identifies one or more text candidates corresponding to a natural 
input object from a natural input recognition dictionary. For example, If the natural input object is a handwriting input, 

35 the recognition module 213 will identify text candidates in a handwriting recognition dictionary that have handwriting 
stroke characteristics matching the handwriting input. Similarly, if the natural input object Is a speech input, the recog- 
nition module 213 will identify text candidates in a voice recognition dictionary that have phonemes matching the speech 
input. The recognition module 213 will then select or recognize the text candidate most closely corresponding to the 
natural Input object. It should be noted that a handwriting recognition dictionary and a voice recognition may optionally 

40 be combined into a single recognition dictionary. 

[0050] For example, if the natural input object Is the handwritten word "text," the recognition module 213 may Identify 
a list of words in the recognition dictionary that con^espond to the shape of the handwritten letters, such as the words 
"test," "tat," "best," "tux," "tent," and "tot." The recognition module 213 then assigns a ranking to each of these text 
candidates based upon its co^espondence to the natural Input, and selects the highest-ranked text candidate to be 

45 the text recognized from the natural Input. Thus, the recognition module 213 does not typically identify only a single 
word corresponding to a natural input object, but instead identifies a list of alternate text candidates in addition to the 
selected text candidate recognized from the natural input. 

[0051] As discussed above, the recognition module 213 may format the selected'text candidate to provide spacing, 
capitalization, punctuation or other formatting. As previously noted, the recognition module 213 may select a text can- 

50 didate or fomnat a recognized text candidates according to heuristics generated by the recognition context module 211 . 
Once the recognition module 213 has generated recognized text from an actual input object, it then forwards the 
generated text to the target application 207 through the text services framework module 205. The recognition module 
213 also forwards the generated text to the object model module 215, along with the original natural input object and 
the text candidates matching the original natural input object. 

55 [0052] Referring now to the object model module 21 5, the object model module 21 5 collects the text generated by 
the recognition module 213, and the original natural input object from which the text was recognized. For example, If 
the original natural Input object was handwriting input, then the object model module 215 may store the various char- 
acteristics of the strokes making up the handwriting input. In addition, the object model module 215 collects the non- 
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selected text candidates Identified by the recognition module 213 as matching thei original natural Input object. By 
collecting this Information for each natural Input object recognized by the recognition module 21 3 for Insertion into the 
target application 207, the object model module 215 creates an object model describing the history of the text recog- 
nition for the target application 207. 

5 [0053] The object model created by the object model module 21 5 can advantageously be used to Improve the user's 
recognition experience by allowing for easier correction of inaccurately recognized natural input. For example, if the 
recognition module 21 3 cannot clearly distinguish between two or more text candidates for a sequence of natural input 
objects, the object model can be used to create a lattice of the alternate text candidates. The lattice may be arranged 
using, for example, a natural language model orother known technique for correlating text. If the user then subsequently 

10 corrects inaccurately recognized text by selecting one of the alternate text candidates In the lattice, the relational 
connections defined by the lattice can be used to connect other inaccurately recognized text. Moreover, if a handwriting 
input object does not con-espond to text, but is instead a drawing, then the lattice fonned with the object model can be 
used to accurately recognize related handwriting input as drawings. For example, a language model might identify a 
drawing by searching for handwriting input objects whose text candidates do not correspond to a sentence that follows 

15 sentence-ending punctuation. 

[0054] Also, the object model can be employed by the user to correct inaccurately recognized text. If the user selects 
multiple words for correction, the object model can be employed to provide the user with combinations of alternate text 
candidates corresponding to the selected words. For example, if the spoken words "hello world" are incorrectly recog- 
nized as "halo walled," then the object model can be employed to provide the user with combinations of the top alternate 

20 text candidates for each inaccurately recognized word. Thus, the user might be presented with the eight combinations 
"halo world," "halo whirled," "hello walled," "hello world," "hello whirled," "yellow walled," "yellow world," and "yellow 
worid." The user could then select the correct combination of alternate text candidates. 

[0055] With some embodiments of the Invention, the object model module 21 5 provides the object model to the target 
application 207, such as by way of the text services framework module 205, for example. The object model can then 

25 be stored by the text sen/ices framework module interface 209 in the target application 207. This conveniently allows 
the target application 207 to employ its own natural language model, if available, to improve the correction of recognized 
text as described above. By storing the object model in the target application 207, the object model can be employed 
even after the target application 207 is no longer communicating with the natural input recognition module 203. Further, 
because the object model includes the original natural input object, a user can subsequently replace recognized text 

30 with the original input object from which it was recognized. 

[0056] As previously mentioned, the natural Input recognition module 203 also includes a correction module 217. 
The correction module 21 7 provides a correction Interface through which a user can con'ect text, as will be discussed 
In detail below. Moreover, if the user employs the correction interface to submit a natural Input object for replacing text 
that already exists In the target application 207, the correction module 217 will notify the recognition context module 

35 211 of the existing text that Is being replaced. As previously discussed, the recognition context module 211 can then 
generate heuristics that will prevent the recognition module 21 3 from generating text identteal to the existing text being 
replaced. 

Operation Of The Recognition Tool 

40 

[0057] The operation of the natural input recognition module 203 will now be described with reference to the flow- 
charts shown in Figures 3A and 3B. First, in step 301 , the user designates an insertion location in the target application 
207. As will be appreciated by those of ordinary skill in the art, the insertion location may be a single Insertion point, 
or It may be a selection of existing text. Next, In step 303, the recognition context module 21 1 obtains context infonnation 

45 regarding the insertion location through the text services framework module 205. The recognition context module 211 
then generates heuristics for controlling the recognition of the natural Input objects In step 305. Next, In step 307, the 
tablet input panel 201 receives a natural input object, which it fonvards to the recognition module 213. 
[0058] In step 309. the recognition module 21 3 generates recognized text from the natural input object based upon 
the heuristics provided by the recognition context module 211. As previously noted, the recognition module 213 may 

50 fonnat the recognized text accorefing to the provided heuristics. In step 311 , the recognition module 213 provides the 
generated text to the object model module 215, along with the original input object and the text candidates Identified 
by the recognition module 213 as matching the original Input object. Then, In step 313, the recognition module 213 
(or, with some embodiments of the invention, the tablet Input panel 201) provides the generated text to the target 
application 207 by way of the text services frameworic module 205. Also, In step 315, the object model module 215 

55 provides the object model to the target application 207 through the text services frameworic module 205. It will be 
appreciated by those of ordinary skill In the art, however, that the recognition module 213 can provide the Infonnation 
contained in the object model directly to the text services framewori< module 205. Lastly, In step 31 7, the target appli- 
cation 207 displays the recognized text received from the recognition module 213. 
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The Correction User hterface 

[0059] As previously noted, the correction module 21 7 may provide a user Interface for correcting text In the target 
application 207. Figure 4 illustrates such a correction Interface 401 according to one embodiment of the Invention. As 

5 seen In this figure, the correction interface 401 includes an alternate list area 403, and an original input object area 
405. The correction interface 401 also includes a command menu area 407 and a selected text area 409. 
[0060] If the text being corrected was inaccurately recognized from a natural Input object, then the alternate list area 
403 will include alternate text choices 41 1 that also correspond to the natural input object. IVIore particularly, as explained 
In detail above, when recognizing text corresponding to a natural Input object, the recognition module 213 may addl- 

10 tionaliy determine a list of one or more alternate text choices that also conrespond to the natural input object. Thus, 
the alternate list area 403 displays the attemate text choices 41 1 that were the next highest-ranked by the recognition 
module 213 below the selected text. If the user detemilnes that an alternate text choice 411 accurately con-esponds 
to the natural input object, the user may select that correct altemate text choice 411 . In response to the user selecting 
one of the alternate text choices 411 displayed In the altemate list area 403. the correction module 217 replaces the 

IS existing text with the selected alternate text choice 411 . 

[0061] As illustrated in Figure 4, the alternate list area 403 displays only five altemate text choices 411 , regardless 
of the total number of altemate text choices identified by the recognition module 213 as possibly corresponding with 
the natural input object. By limiting the number of displayed alternate text choices 41 1 to a small value, such as a value 
In the range of 5 to 9, the correction Interface 401 significantly reduces the amount of reading time required by the user 

20 to identify the accurate altemate text choice 411 . At the same time, by providing more than just one or two altemate 
text choices 411 , the correction Interface 401 substantially Increases the probability that the accurate altemate text 
choice 411 will be displayed in the alternate list area 403. 

[0062] It should also be noted that the initial alternate text choice 411 displayed in the altemate list area 403 may 
always be the reverse capitalization of the existing text. With handwriting, a handwriting recognition process may have 
25 difficulty distinguishing the lower-case form of a letter from its upper-case form. Similarly, a voice recognition process 
usually will be unable to distinguish between text that should begin with an upper case letter and text that should begin 
with a lower-case letter. Thus, by providing the reverse capitalization of the existing text as the first alternate text choice 
41 1 displayed in the alternate list area 403, the correction interface 401 allows a user to very quickly con'ect this common 
type of recognition error 

30 [0063] With some embodiments of the invention, the alternate text choices 41 1 may be displayed in the alternate list 
area 403 based upon the confidence that the recognition module 211 has in their accuracy. For example, alternate text 
choices 411 for which the recognition module 211 has a high level of confidence can be displayed in green, while 
altemate text choices for which the recognition module 21 1 has a tow-level of confidence can be displayed in red. Of 
course, the alternate list area 403 can display the altemate text choices 41 1 according to a variety of different confidence 

35 levels. Techniques for detemnining such confidence levels are disclosed in the copending U.S. Patent Application No. 
10/120,163, filed April 9, 2002, entitled "Assignment And Use Of Confidence Levels For Recognized Text," naming 
Manish Goyal et al. as inventors, which application is incorporated entirely herein by reference. 
[0064] The original input object area 405 displays the original input object from which the inaccurate text was rec- 
ognized, in the example shown in Figure 4, the original input object was the handwritten word "text." This conveniently 

40 allows the user to review the original input object before correcting the inaccurate text. With some embodiments of the 
invention, the original input object area 405 may also serve as a command button for replacing the inaccurate text with 
the original input object. Thus, with these embodiments, activating the original input object area 405 would replace the 
Inaccurate text shown In the target application 207 with the hand written word "test" as displayed and the original input 
object area 405. If the original input object was a voice input, then the original input object area 405 could be imple- 

45 mented as a button that, when activated, audibly played back the original voice input object. 

[0065] The command menu area 407 displays various commands for editing the inaccurate text. For example, in the 
illustrated embodiment, the command menu area 407 Includes the command "delete" 41 3, which the user may activate 
to delete the inaccurate text. The command menu area 407 also Includes the command "re-write/speak" 415. When 
the user activates this command 413, the connection module 217 summons the tablet input panel 201, and transfers 

50 focus from the command menu area 407 to the user interface provided by the tablet Input panel 201 . The user can 
then subsequently write or speak the desired text employing the tablet input panel 201 , and the recognition module 
213 will recognize the text to replace the Inaccurate text. As previously noted, however, when the user activates the 
command re-write/speak 415, to submit natural Input to replace the inaccurate text, the recognition context module 
211 will ensure that the recognition module 213 does not select a text candidate Identical to the inaccurate text. 

55 [0066] The command menu area 407 further includes the command "ignore" 417. This command may be used when, 
for example, the target application 207 has indicated that the inaccurate text is misspelled or grammatically Incoaect, 
and the user disagrees with the application's assessment. In addition, the command menu area 407 includes the 
command "add to dictionary" 41 9. Again, this command may be used when the designated text is not actually inaccu- 
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rate, but the target application 207 has Indicated that the existing text Is misspelled, and the user disagrees with the 
application's assessment. This command adds the existing text to the recognition dictionary employed by the recog- 
nition module 211. it should be noted that, where the recognition module 211 supports both voice and handwriting 
recognition, activating the command 219 may add the existing text to both the recognition dictionary for the voice 
5 recognition process and the recognition dictionary for these handwriting recognition process. Alternately, the command 
419 may add the existing text to a recognition dictionary shared between the voice recognition process and the hand- 
writing recognition process. 

[0067] As seen in Figure 4, the selected text area 409 displays the inaccurate (or otherwise designated) text. The 
user may conveniently employ the selected text area 409 to search for and display an accurate altemate text choice 

10 411 In the altemate list area 403, More particularly, the user may select one or more characters of the text displayed 
In the selected text area 409. Alternately, as shown In Figure 5, the user may select only a portion of the existing text 
displayed In the target application 207. In response to the selection, the recognition module 21 3 will provide only those 
text candidates it identified that correspond to the selected text. Thus, the recognition module 213 may identify only 
text candidates that differ by the selected characters. For example as shown In Figure 5, when the characters "es" are 

15 selected, the recognition module 211 only provides the correction module 217 with the alternates "tat," 
tent," "tot," "teat," and "tort." The correction Interface 401 thus omits the alternates "best" and "tux" shown in Figure 
4." Thus, the selected text area 409 provides the user with a powerful tool for allowing the recognition module 213 to 
quicl^iy eliminate alternate text choices 411 that contained inaccurately recognized characters. 

20 Use Of The Recognition Tooi Wittiout The Text Services Ramewortc Moduie 

[0068] In the above-described embodiments of the Invention, the recognition context module 211 obtains context 
Infomiation from the target application 207 through the text services framewori< module 205. As will be appreciated by 
those of ordinary sl<ill in the art, however, not all applications will support the use of the text services framework module 

25 205. With these target applications 207, the recognition context module 211 can instead obtain context infomiation 
regarding the target application 207 from Its edit controt/or rich edit control objects 601 or other data stmctures, as 
previously noted. An illustration of this configuration, omitting the text services framewori< module 205, is shown In 
Figure 6. As l<nown in the art, an edit control is a rectangular control window typically used in a dialog box to permit 
the user to enter and edit text. Similarly, a rich edit control is a window in which the user can enter, edit, fomat, print, 

20 and save text, which can be assigned character and paragraph fomiatting, and which can include embedded COM 
objects. The characteristics of edit controls and rich edit controls In the target application 207 will typically be stored 
in tables that can be accessed by the recognition context module 211 . 

[0069] For example, if the user Is attempting to insert recognized text into the address field of the Microsoft Internet 
Explorer 5 browser application, the recognition context module 211 will detemiine that the software application does 
35 not support the use of the text services framewori< module 205. in response, the recognition context module 211 can 
look up the characteristics of this fietd in the appropriate table of the browser application, and thereby detemiine that 
this control is a universal resource locator (URL) address bar. The recognition context module 211 will then generate 
heuristics for the recognition module 213 requiring that generated text being Inserted into this field have no spaces. 

40 Conclusion 

[0070] Although the invention has been defined using the appended claims, these claims are exemplary in that the 
invention may Include the elements and steps described herein in any combination or sub combination. Accordingly, 
there are any number of alternative combinations for defining the Invention, which incorporate one or more elements 

<5 from the specification. Including the description, claims, and drawings, in various combinations or sub combinations. 
For example, while the natural input recognition module of the invention is shown as a distinct entity, one more com- 
ponents of this moduie may be incorporated and the tablet input panel 201 . Also, while the recognized text is described 
above as being corrected In the target application 207, if the tablet input panel 201 provides a stage for displaying 
recognized text before it Is delivered to the target application 207. then the recognized text may be corrected at the 

50 stage rather than in the target application 207 as described above. 

[0071 ] it will be apparent to those skilled in the relevant technology, in light of the present specification, that alternate 
combinations of aspects of the Invention, either alone or In combination with one or more elements or steps defined 
herein, may be utilized as modifications or alterations of the Invention or as part of the Invention. It may be Intended 
that the written description of the invention contained herein covers a!! such modifications and alterations. For instance, 

55 in various embodiments, a certain order to the data has been shown. However, any reordering of the data is encom- 
passed by the present invention. Also, where certain units of properties such as size (e.g., in bytes or bits) are used, 
any other units are also envisioned. 
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Claims 

1 . A method of generating text from a natural Input object for insertion Into a target application, comprising: 

5 receiving a natural Input object from which recognized text Is to be recognized; 

obtaining a context for insertion of recognized text recognized from the natural Input object Into a target ap- 
plication; and 

10 generating recognized text from the natural Input object based upon the obtained context. 

2. The method of generating text recited In claim 1 , wherein the context Is obtained from the target application. 

3. The method of generating text recited in claim 2, wherein 

IS obtaining a context for Insertion includes perfomilng one or more of: 

identifying a field type for a field in the target application Into which the recognized text will be Inserted; 
identifying text that has already been Inserted Into the field; 

20 

identifying text adjacent to an insertion location for inserting the recognized text; 
Identifying a length of the insertion location for inserting the recognized text; and 
25 identifying any text that will be replaced by the recognized text. 

4. The method of generating text recited in claim 3, wherein generating text from the natural input object Includes 

recognizing text from the natural input object, and 
formatting the recognized text. 

30 

5. The method of generating text recited in claim 4, wherein fonnatting the recognized text includes providing spacing 
for the recognized text. 

6. The method of generating text recited in claim 5, further comprising: 

35 

assigning a category to the field in the target application into which the recognized text will be Inserted; and 

providing spacing for the recognized text according to the category for the field In the target application into 
which the recognized text will be inserted. 

40 

7. The method of generating text recited In claim 6, further Including: 

detemninlng if context information is provided for the field in the target application Into which the recognized 
text will be Inserted; and 

45 

If context infomiation is not provided for the field in the target application into which the recognized text will 
be inserted, then categorizing the field into a first category. 

8. The method of generating text recited in claim 7, further including 

so determ Ining if the context Infomtation allows for control of insertion of text into the field in the target application 

Into which the recognized text will be inserted; and 

if the context infomiation allows for control of insertion of text into the field in the target application into which 
the recognized text will be inserted, then categorizing the field into a second category. 

55 9. The method of generating text recited In claim 6, further Including: 

providing spacing for the recognized text based upon a language of the recognized text. 
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10. The method of generating text recited In claim 5, further comprising: 

detennining if text before the insertion location fomris a word when concatenated with the recognized text; and 
If text before the insertion location fomris a word when concatenated with the recognized text, omitting a space 
s before the recognized text. 

11. The method of generating text recited In claim 5, further comprising: 

determining If text after the insertion location fomns a word when concatenated with the recognized text; and 

10 

if text after the insertion location fomns a word when concatenated with the recognized text, omitting a space 
after the recognized text. 

12. The method of generating text recited In claim 4, wherein fomnattlng the recognized text includes detemilnlng 
15 capitalization for the recognized text. 

13. The method of generating text recited in claim 12, wherein fomiatllng the recognized text includes capitalizing 
every character in the recognized text. 

20 14. The method of generating text recited In claim 12, wherein fomnattlng the recognized text Includes retaining a 
capitalization case, recognized from the natural input object case, in the recognized text. 

15. The method of generating text recited In claim 4, wherein fomnattlng the recognized text Includes detemnlning 
punctuation for the recognized text. 

25 

16. The method of generating text recited In claim 15, fomnattlng the recognized text Includes adding punctuation to 
the recognized text. 

17. The method of generating text recited in claim 3, wherein generating text from the natural input object Includes 
30 detennining if the recognized text Is replacing text existing in the target application; and 

if the recognized text is replacing text existing in the target application, recognizing the recognized text from 
the natural input object to be different from the existing text being replaced, 

18. The method of generating text recited In claim 1, wherein the context is obtained from a correction process for 
35 correcting existing text. 

19. The method of generating text recited in claim 18, further comprising: 

detemnlning of the recognized text is replacing existing text; and 

40 

if the recognized text is replacing text existing In the target application, recognizing the recognized text from 
the natural input object to be different from the existing text being replaced. 

20. A method of inserting text recognized from a natural input object into a target application, comprising: 

45 

determining one or more text candidates to which the natural Input object conresponds: 

recognizing text from the natural input object by selecting the text candidate that most closely corresponds to 

the natural input object; 

50 

Inserting the text object into a target application; and 

providing the target application with Infomnation relating to the natural Input object. 
S5 21 . The method of Inserting text recited in claim 20, further comprising: 

providing the target application with a plurality of the one or more text candidates. 
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22. The method of inserting text recited in claim 20, further comprising: 

providing the target application with the natural Input object. 

5 23. The method of Inserting text recited in claim 20, further comprising: 

providing the target application with both the natural Input object and a plurality of the one or more text can- 
didates. 

10 24. A method of recognizing text from natural input objects, comprising: 

determining one or more first text candidates to which a first natural Input object con^esponds; 
determining one or more second text candidates to which a second natural Input object con-esponds; 

IS 

creating a lattice of possible text combinations from the first text candidates and the second text candidates; 

receiving Input selecting one of the first text candidates as recognized text; and 

20 in response to receiving Input selecting one of the first text candidates as recognized text, employing the lattice 

to select one of the second text candidates as recognized text. 

25. The method of recognizing text from natural input objects recited in claim 24, further comprising: 
25 receiving from a recognition process: 

first text corresponding to the first natural input object; 
the one or more first text candidates; 

30 

second text con^esponding to the second natural Input object; and 

the one or more second text candidates; 

35 changing the first text corresponding to the first natural input object to the selected one of the first text candi- 

dates; and 

changing the second text corresponding to the second natural Input object to the selected one of the second 
text candidates. 

40 

26. A method of correcting text, comprising: 

receiving Input designating existing text; 

45 receiving a natural input object for replacement of the existing text; 

determining one or more text candidates corresponding to the natural Input object, such that none of the text 
candidates Is the same as the existing text; 

50 selecting one of the one or more text candidates that most closely con-esponds to the natural object; and 

replacing the existing text with the selected text candidate that most closely corresponds to the natural object. 

27. The method of correcting text recited in claim 26, further comprising: 



55 



displaying a correction interface for connecting the existing text, the correction interface Including an object for 
inputting the natural input object to replace the existing text; 
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receiving input selecting the object for Inputting the natural Input object to replace the existing text; and 
providing an Interface for Inputting the natural Input object. 

28. A method of correcting existing text, comprising: 

receiving input designating existing text and selecting only a portion of the existing text; 

selecting one or more text candidates con-espondlng to the selected portion of the existing text; 

displaying the one or more text candidates; 

receiving input selecting one of the one or more text candidates; and 

replacing the existing text with the selected text candidate. 

29. The method of correcting existing text recited in claim 28, wherein each of the one or more candidates differ from 
the existing text by only the selected portion of the text. 

30. The method of con-ecting existing text recited in claim 28, wherein each of the one or more text candidates includes 
the selected portion of the text. 

31. The method of correcting existing text recited in claim 28, wherein the one or more text candidates are selected 
from a recognition dictionary. 

32. A recognition tool for recognizing text from natural Input objects, comprising: 

a recognition context module that detemnines a context of an insertion locating into which recognized text will 
be Inserted; 

a recognition module that 

identifies one or more text candidates corresponding to a natural input object; 

selects one of the one or more text candidates that most closely corresponds to the natural input object; 

and 

generates text from the selected text candidate based upon the context detemnined by the recognition 
context module. 

33. The recognition tool recited In claim 32, wherein the recognition module selects the one of the one or more text 
candidates based upon the context detenmlned by the recognition context module. 

34. The recognition tool recited In claim 32, wherein the recognition module formats the selected text candidate based 
upon the context determined by the recognition context module. 

35. The recognition tool recited In claim 32. wherein the recognition module formats the selected text candidate based 
upon the context determined by the recognition context module. 

36. The recognition tool recited in claim 35, wherein the recognition module provides spacing for the selected text 
candidate based upon the context detemiined by the recognition context module. 

37. The recognition tool recited In claim 35, wherein the recognition module determines capitalization for the selected 
text candidate based upon the context detenmlned by the recognition context module. 

38. The recognition tool recited In claim 35, wherein the recognition module detemnines punctuation for the selected 
text candidate based upon the context determined by the recognition context module. 

39. The recognition tool recited in claim 32, further comprising: 

an object model module that, for each of a plurality of selected text candidates, stores 
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the natural Input object for which the selected text candidate was selected; and 

the one or more text candidates corresponding to the natural Input object for which the selected text 

candidate was selected. 

5 40. The recognition tool recited in claim 32, further comprising: 

a con-ection module that provides a user interface for correcting text generated by the correction module. 

41. The recognition tool recited In claim 40, wherein 
10 if the correction module receives a natural input object to replace existing text, the correction module notifies 

the recognition context module of the existing text that is being replaced; and 

the recognition context module prevents the recognition module from generating text identical to the existing 
text that is being replaced. 

15 
20 
25 
30 
35 
40 
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